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ABSTRACT 



In a compressed domain digital communications system, a 
method for reducing a variable latency associated with a 
buffer and at least partially resulting from at least one splice 
between a FROM bitstream and a TO bitstream each includ- 
ing data corresponding to a plurality of frames, the method 
including: selectively deleting data corresponding to a select 
at least one of the frames from the buffer based upon the 
variable latency so as to reduce the variable latency when an 
amount of data corresponding to a number of frames present 
in the buffer is greater than a given number of frames; and, 
regulating a flow of data in the system to prevent an 
underflow condition in the system by effecting a repeat last 
frame command and prevent an overflow condition in the 
system by slowing a rate of transmission for the data 
associated with at least one of the frames in the TO bit- 
stream. 

20 Claims, 3 Drawing Sheets 



INPUT DATA STREAM 



S4P 304i 



S*P PCR8 — |S9 3)2 



PES, SEQ, PIX 
AND VBV DELAY 
DECODE 


FR RATE 
DECODE 




PCRBSP 
CONVERTER 


PTS 
DECODE 




DTS 


DECODE 


S3f0l 306 


S3011 






06/08/2004, EAST version: 1.4.1 



U.S. Patent Dec. 11,2001 Sheet 1 of 3 US 6,330,286 Bl 




06/08/2004, EAST version: 1.4.1 



U.S. Patent Dec. ll, 2001 Sheet 2 of 3 US 6,330,286 Bl 



o 

CVJ 

cc 
O 

CO 
CO 
LU 
O 

o 

CC 
Q. 

O 

3 § 

p p 



Q 5 
uj O 



cc 

LU 
X 

p 



o 

_l 

< 
> 

CO 

> 



erg 

CD O 



V V 



o 

CO 
CM 



Q 
CO 



x o 



> 

CO 



< 

»- LU 

Q_ 



35 1 



I 



o 

CO 
CO 
CO 



CO 



1 



CC 
UJ 

—I 

-J 

o 

cc 



o 
o 



> 
co 



CVJ 
CO 



o 

CO 

cc 

LU 

CO 

I— CO 



8 



> 

CO 

CO 



> 

CO 




> 




eg 




CO 








o 
o 




-J LU 


o 





o 

CM 
CVJ 



Ol 
O 

evil 



Q: 



O 
CVJ 



06/08/2004, EAST Version: 1.4.1 



U.S. Patent Dec. ll, 2001 Sheet 3 of 3 US 6,330,286 Bl 




06/08/2004, EAST version: 1.4.1 



US 6,330,286 Bl 

1 2 

FLOW CONTROL, LATENCY CONTROL, cile differences between the input FROM and TO bitstream 

AND BITRATE CONVERSIONS IN A TIMING rates and the output stream rate. This feature provides for 

CORRECTION AND FRAME asynchronous inputs from remote or local sources. 

SYNCHRONIZATION APPARATUS Output stream flow control is provided to transition the 

5 output stream buffer levels and VBV delays to the values in 

RELATED APPLICATIONS tne j 0 stream. This permits the switch to replicate previously 

This application claims the benefit of U.S. Provisional generated variable bit rate (VBR) stream data rates after a 

Application No. 60/138,223, entitled "FLOW CONTROL, transitl0n P eri od from one stream to the next. It should be 

LATENCY CONTROL AND BITRATE CONVERSIONS understood that the ability to utilize VBR streams is impor- 

IN A TIMING CORRECTION AND FRAME SYNCHRO- 10 tant t0 studlos since thev P rovide broadcasters with channel 

NIZATION APPARATUS", filed on Jun. 9, 1999, the entire s P ace ' whlch can of course be used for opportunistic data 

disclosure of which is hereby incorporated by reference. channels that provide additional sources of revenue. 

The present invention is also related to commonly However, it has been recognized that adding and dropping 

assigned U.S. patent application Ser. No. 08/864,326, 15 frames for buffer control essentially changes the end-to-end 

entitled "TIMING CORRECTION METHOD AND delav ' or latencv > of me overa11 system. More particularly, 

APPARATUS", filed May 28, 1997 (the "'326 application) when the remote clock fre q uenc y * consistently higher than 

and U.S. patent application Ser. No. 08/864,325, entitled the local clock frequency, the end-to-end system delay 

"METHOD AND APPARATUS FOR INFORMATION ^creases. When the remote clock frequency is consistently 

STREAM FRAME SYNCHRONIZATION", filed May 28, 20 lower than the local clock frequency, the end-to-end system 

1997 (the '"325 application), now U.S. Pat. No. 6,061,399 delav Ureases. When the current value differs from the 

issued on May 9, 2000, each of which are also incorporated stored value for end-to-end delay by a value equal to one 

herein by reference in their respective entireties. frame ' and other exteraal criteria are met, a frame is either 

dropped or added. As the buffer of the next output, e.g. 

GOVERNMENT INTERESTS corresponding to a preselected or TO stream, fills while 

25 awaiting a proper exit point in the FROM or current stream, 

This invention was made with U.S. government support there is the potential for this additional unpredictable change 

and funded by National Institute of Standards and Technol- i n end-to-end delay or latency. This is undesirable, and there 

ogy (NIST) under contract No. 70NANB5H1174. The U.S. is a need to eliminate this unpredictable and variable latency. 

government has certain rights in this invention. 1t ■ u - ♦ c *u * • j i . 

& ^ 30 It is an object of the present invention to provide latency 

BACKGROUND OF INVENTION control whereby a variable latency generated by switching 

or transitioning between compressed data sources is reduced 

The referenced '325 and '326 applications cited and incor- and systematically eliminated, 
porated by reference herein address timing correction and 

frame synchronization in a splicer/switcher for bitstreams. 35 SUMMARY OF INVENTION 

The goal of splicing compressed bit streams is to change or T , , . ,. . 4 , ... 

« - t . c jj, . i In a compressed domain digital communications system, 

transition from one compressed data source to a second , c j • • . , , • , . , 

A* *u * i a- • *u a method for reducing a variable latency associated with a 

compressed data source with minimal disruption in the , rr , . Jf- n u- t ♦ i * i- 

Aa „Z*<>A u-i •« • ■ u-* * r butter and at least partially resulting from at least one splice 

decoded program, while mamtaimng bitstream compliance , t „,_ CDm .^ 4 3 , . , . , , 

prior to, through and after the transition. Input streams are 40 ^1 * b . l . tS,re , am \™ b ;«f eam ««* 

de-multiplexed to the Packetized Elemental Stream (PES) 40 mg . d f ° a P 1 "^ °* *m ** ***** 

level before being processed by individual elementary T^LTSTl T da " C0 "Tf ng /° aSel f f 

stream type process^. Program vkleo and audio are spliced at l ™ °f ° f ^ ^ buffer based upon the 

. „ - t x . 5 • u • j v • • variable latency so as to reduce the vanab e latency when an 

at access unit (frame) boundaries, however video splicing is n , fJ / , cc 

„ 11 u- u • • ** j * . . . , . amount of data correspondmg to a number of frames present 

typically higher in priority than audio sphcing which is • fUA . ~ • , ■ . uiiva^m 

t ;,„- «/ , A r J . . . 6 T - 4 5 in the buffer is greater than a given number of frames; and, 

typically dependent upon video splice timing. Up sync _ , fl 6 f , , . e 4 t ' * 

/fl -j j j- • ■ * ■ J (L u iL regulating a flow of data in the system to prevent an 

between program video and audio is maintained through the „„. « 6 . 4 , J „ y^y^au 

™r™ # a a r *■ * j j j underflow condition in the system by effecting a repeat last 

splice point. A continuous flow of time stamped video and ^ , , \ * F 7 , 

audio frames, without timing discontinuities, is maintained hvTwino a f D , ™ ™t>Z "J*? 

in the output stream. <n •J**?' b J s f° wln S a / ate of f transm^.on for the data 

r 50 associated with at least one of the frames in the TO bit- 

The video splicing approach described in the referenced stream 
and incorporated '325 and '326 applications examines 

incoming streams to extract stream parameters that are used BRIEF DESCRIPTION OF THE FIGURES 
to determine stream entry and exit points and calculate 

values required by the outgoing stream. Exit points are 55 Vanous olher ob J ects > features and advantages of the 

found in the current output stream, e.g. a "FROM" stream, invention will become more apparent by reading the fol- 

while entry points are found in the next output stream, e.g. lowin 8 detailed description in conjunction with the 

a "TO" stream. Pre-selection of the TO stream from multiple drawings, which are shown by way of example only, 

possible input streams preferably limits the number of input wherein: 

processing circuits needed to two. 60 FIG. 1 shows a block diagram of an information process- 

A time base correction process restamps the stream m S svstem deluding a frame synchronizer; 

selected for output. A transport stream encoder maintains the FIG. 2 shows a block diagram of a frame synchronizer 

output time base and re-multiplexes the PES streams into an suitable for use in the information processing system of FIG. 

Advanced Television Systems Committee (ATSC) compli- 1; and, 

ant output stream. A frame synchronization feature adds or 65 FIG. 3 shows a block diagram of a FTS-DTS retiming 

drops frames from the output stream as needed to maintain system suitable for use in the information processing system 

Video Buffering Verifier (VBV) buffer fullness or to recon- of FIGS. 1 and 2. 
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DETAILED DESCRIPTION OF THE Briefly, the video retiming unit 300V decodes and retimes 

INVENTION the old Presentation Time Stamps (PTS) and decode time 

This present invention builds upon the teachings of the stamps (DTS) of the video stream using new timing infor- 

referenced and incorporated 325 and 326 applications by mation derived from a Program Clock Reference Base 

enhancing the concept of frame synchronization by intro- 5 (PCRB) signal S9. Audio retiming unit 300A retimes the old 

ducing latency control and further refining the concept of PTS of tDe audio stream using new timing information 

flow control, Bitrate conversion from constant bitrate (CBR) derived from a Program Clock Reference Base (PCRB) 

types of encoding to variable bitrate (VBR) encoding is also si S nal S9 - ^ ^ deo arjd audio retiming units 300V, 300A 

provided according to the present invention, and has great produce, respectively, a retimed video stream S7PV and a 

value to broadcasters and other users of the system. 10 re ^ med au dio stream S7PA. In addition, the video retiming 

More particularly, and referring now to FIG. 1, therein is uait 300 V generates a number of control signals. The buffer 

illustrated a transport layer switching and retiming system A° w control signal S12 is used to regulate data production 

100 according to the commonly assigned 325 application. in the video frame synchronizer 200 and the PES source 140. 

Therein, a transport stream decoder 110 decodes an input The buffer flow control signal S12 is depicted in FIG. 1 as 

transport stream SUN that is asynchronous to a 27 Mhz 15 being generated by the video PTS-DTS retiming unit 300V 

studio reference signal S10 from station clock 160, illustra- During an initialization process in the Video PTS-DTS 

tively a remote feed, to produce a program stream SP. The retiming circuit 300V, the VBV delay of the VBV buffer is 

program stream SP comprises a plurality of Packetized aligned to the VBV delay of the incoming stream. A signal 

Elementary Streams (PES). A PES demultiplexer 120 S330, originating in Video PTS-DTS retiming circuit 300V 

demultiplexes the program stream SP to produce the plural- 20 indicates the VBV delay of the incoming stream is equal to 

ity of PES streams including an audio stream S1A and a the VBV delay of the VBV buffer at the end of the alignment 

video stream S1V Those skilled in the art will recognize that process. The number of frames in the combined VBV buffer 

PES streams are embedded within the transport packet and the input FIFO of the video frame synchronizer 200 is 

structure, therefore the following descriptions will deal only stored and used to determine whether to add or drop frames, 

with PES streams. The PES streams S1A, S1V include 25 Alternately, when transport streams are being received, the 

timing information such as Presentation Time Stamps (PTS) PCR offset (the difference between the incoming PCR and 

and decode time stamps (DTS) that are asynchronous with the local PCR) can be used to determine whether to add or 

respect to the 27 MHz station clock S10. The video and drop frames. The audio retiming unit 300A does not include 

audio PES streams SI V, S1A are respectively coupled to buffer flow management functions but otherwise operates in 

video frame synchronizer 200 and audio processor 201. 30 substantially the same manner as the video retiming unit 

The video frame synchronizer 200 will be described more 300V, and therefore will not be discussed further. Apparatus 

fully below with respect to FIG. 2. Briefly, the video frame suitable for use as the video retiming unit 300V is more fully 

synchronizer 200 operates to ensure that the received system described regarding FIG. 3. 

data rate is adjusted to match the local output data rate. The A transport stream encoder (TSE) 150 receives the 

frame synchronizer monitors a buffer flow control signal 35 retimed video S7PV and audio S7PA PES streams and 

S12 that is indicative of buffer utilization in down stream encodes these streams to produce an output transport stream 

system components, and in response thereto increases or Sll. TSE 150 includes a PCR generator and retimer, 

decreases the data rate of the video frame synchronizer 200 receives the 27 MHz station clock signal S10 and performs 

output stream S2PV to maintain an appropriate level of PCR stamping of the output transport stream Sll. The PCR 

buffer utilization. A decision to add or drop frames is based 40 generator also produces the synchronized PCR base refer- 

upon a change in the number of frames in the receiver ence signal S9 at a 90 Khz rate (the station clock of 27 MHz 

circuitry over time. The decision to add or delete video divided by 300). The PCRB reference signal S9 is fed back 

frames is coupled as signal SD to audio processor 201 so that to the retiming units 300V, 300Afor use in the PTS and DTS 

appropriate audio frames may be correspondingly added or stamping process. 

deleted. The video frame synchronizer 200 and audio pro- 4S The above-described information processing system 100 

cesser 201 produce respective frame-synchronized video may be used in a digital communication system, such as the 

S2PV and audio S2PA Packetized Elementary Streams system described in the Advanced Television Systems Com- 

(PES) which are coupled to a PES switcher 130. In addition mittee (ATSC) digital television (DTV) standard document 

to the buffer management function, the video frame syn- A/53, also incorporated herein by reference. In such a 

chronizer 200 aligns the video frames of a stream prior to 50 system it is often desirable to switch from a first (i.e., the 

selection of that stream by the PES switcher 130, such that currently-selected or from) video and/or audio source to a 

the first two frames provided by the video frame synchro- second (i.e., the next-selected or to) video and/or audio 

mzer 200 are anchor frames, such as an 1-frame and a S0U rce in a smooth or "seamless" manner (i.e., minimal 

P-frame. undesirable visual or aural artifacts). 

PES switcher 130 also receives a second pair of video 5 5 To ensure a smooth switching between MPEG streams the 

S3PV and audio S3PA Packetized Elementary Streams from following processing operations are deemed to be desirable, 

a PES source 140 (e.g., a digital video disk player, tape First, the next-selected stream should be subjected to a frame 

player, camera, server or the like). The PES source 140 alignment process prior to switching. The frame alignment 

monitors the buffer flow control signal S12 that is indicative process ensures that the first frame of the next-selected 

of buffer utilization in downstream system components and, 6 0 stream that is coupled to the output is a non-predicted frame 

in response thereto, analogously increases or decreases the i.e., an intraframe or "I-frame." The second frame should be 

data rate of the PES source 140 output streams S3PV, S3PA another I-frame or a frame predicted using only the first 

to maintain an appropriate level of buffer utilization. frame, i.e., a forward predicted or "P-frame." When an 

In response to a control signal (not shown) the PES I-frame is not followed by another I-frame or a frame 

switcher 130 selects and couples one pair of video and audio 65 predicted using only the first frame, i.e., a forward predicted 

Packetized Elementary Streams S4PV, S4PA to respective or "P-frame," a specially coded P-frame can be added 

video and audio retiming units 300V, 300A. between the previously existing I-frame and B-frame series 
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or intervening B-frames can be dropped to close the Group 
Of Pictures (GOP). This artificially inserts an entry point 
into the stream during the alignment process. The frame 
alignment process is performed by the frame synchronizer 
200. Second, the VBV delay alignment is performed by the 5 
PTS-DTS retiming unit 300V. Alignment results are indi- 
cated on signal S330 of FIG. 1. Third, the VBV delays and 
the utilization of buffer memory (e.g., far-end decoder input 
buffer or FIFO 220) should be monitored before and after 
switching. After S330 indicates VBV delay alignment, an 
increase or decrease in the number of frames stored in the 
combined VBV-buffer and the input FIFO in the frame 
synchronizer 200 determines the need to add or drop frames. 

Referring now also to FIG. 2, it shows a block diagram of 
an illustrative frame synchronizer suitable for use in the 
information processing system of FIG. 1. The frame syn- 15 
chronizer 200 receives a Packetized Elementary Stream 
(PES) S1V, illustratively a video stream, aligns the received 
PES stream in a first operating mode (i.e., a "pre-select" 
mode) and couples the PES stream to an output in a second 
operating mode (i.e., a "selected" or "steady-state" mode). 20 
In addition, the frame synchronizer 200 performs various 
buffer management operations during the second operating 
mode. The controller 230 is optionally forced into the 
pre-select or selected modes of operation by a signal 
SWITCH CONTROL. Moreover, the controller optionally 25 
transmits a signal SWITCH to cause a switching operation 
to take place by a "down-stream" switcher e.g., PES 
switcher 130. 

The input video PES stream S1V is coupled to a clock 
extractor 210, a first-in first out (FIFO) memory 220 and a 30 
controller 230. FIFO 220 is preferably large enough to hold 
an I-frame plus all subsequent frames up to and including the 
next I-frame. The video stream S1V is clocked into FIFO 
220 using a load clock signal S2V associated with the 
arriving data. A FIFO output stream S3V representing the 35 
temporarily stored input video stream S1V is coupled to a 
frame add/drop unit 240. Add/drop unit 240 produces an 
output video stream S2PV comprising the FIFO output 
stream S3V (pass-through mode), the FIFO output stream 
S3V with additional frames added (frame add mode) or 40 
FIFO output stream S3V with frames deleted (frame drop 
mode). The mode of operation of addldrop unit 240 is 
controlled by a signal S6V from the controller 230. The 
controller 230 also monitors FIFO 220 via a monitor signal 
S5V and controls FIFO 220 via a control signal S4V The 45 
controller receives a buffer flow control signal S12 that is 
indicative of buffer utilization in down-stream system com- 
ponents. The controller causes the add/drop unit 240 to add 
or delete video frames, thereby maintaining timing synchro- 
nization of the remote and local sources. 50 

The controller 230 monitors the input video stream S1V 
and the FIFO 220. The controller also controls the operation 
of the FIFO 220 by causing the FIFO to discard all or 
portions of the stored input video stream S1V for example. 
For illustrative purposes it is assumed that each PES packet 55 
contains one video access unit (i.e., video frame). Before 
this video stream can be switched to e.g., by PES switcher 
130 and become the output stream an alignment for switch- 
ing purposes is made. Since the MPEG Sequence Header 
contains information vital to decoding the pictures, a switch 60 
into a stream preferably occurs just prior to the sequence 
header. The last sequence header that has entered the FIFO 
is stored for use when a sequence header does not immedi- 
ately precede an I-frame. The FIFO is therefore flushed of all 
data until an I-frame sits at the output of the FIFO. 55 

The bitstream is constrained such that I -frames are fol- 
lowed by P- frames or I -frames in transmission order to 
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insure replacement of both anchor frames at switch points. 
When I-frames are followed by B-frames, a P-frame com- 
prised of uncoded macroblocks with zero motion vectors 
may be optionally inserted between the I-frame and the 
B-frames or intervening B-frames can be dropped to create 
a closed GOP. This P-frame will act as a repeat last frame 
command. The visual effect of the repeat frame on video 
quality is masked by the scene change that occurs coincident 
with the switching of streams. The buffer flushing process 
continues each time an I-frame enters the FIFO until a 
decision to switch to this stream is made. If the I-frame has 
been immediately preceded by a sequence header, then the 
switching process continues using the PES and sequence 
headers within the PES packet. If the PES packet for the 
I-frame does not contain a sequence header, then the last 
stored sequence header is multiplexed into the stream. The 
resulting output stream contains a sequence header just prior 
to the I-frame. 

At the switch point at least two frames must reside within 
the FIFO memory. This condition permits a calculation of 
the temporal reference at the switch point. Optionally, a 
group of pictures (GOP) header may be inserted prior to the 
output of the I-frame and the temporal reference of the 
I-frame set to zero. Offsets from the incoming temporal 
references and the new values are calculated at the switch 
point. 

Using such a device and method, streams can essentially 
be left at any point with the consequence of the last frames 
in the currently-selected stream possibly being of incorrect 
display order. However, the visual effect will be masked by 
the natural shift to new video material. If the currently- 
selected stream is left just prior to an anchor frame, no frame 
reordering effect will be encountered. An additional number 
of frames, belonging to the currently-selected stream, reside 
in the decoder buffer at this time. The number of frames in 
the decoder buffer have been tracked by a frame byte counter 
in the PES retiming circuit 300V. 

An MPEG picture header contains a Video Buffering 
Verifier (VBV) delay number indicative of the amount of 
time a decoder should wait (after the picture header enters 
the decoder's VBV buffer) before decoding the picture. The 
VBV delay equates to the decode time stamp (DTS) minus 
the program clock reference base PCRB. This equation 
relates to the number of frames in the decoder buffer. The 
VBV delay at any given instant can be calculated using this 
equation at the output of the PES retiming circuit 300V 

The incoming MPEG stream contains the VBV delay 
created when the stream was encoded or retimed. Both the 
next-selected stream (TO stream) and the currently-selected 
stream (FROM stream) VBV delays are known at the time 
of switching. In general these values will not match. Since 
during video encoding it is generally desirable to place as 
many frames in the decoder buffer as possible, the VBV 
delay of the last frame in the currently-selected stream 
should be large. This creates an opportunity to match the 
VBV delay of the next-selected stream to the actual VBV 
delay in the decoder buffer. 

The next-selected stream and currently-selected stream 
VBV delays are compared in the PES retiming circuit 300V 
at the time of switching. This initially indicates whether the 
decoder buffer contains too much or too little delay. When 
too little delay is in the decoder buffer to satisfy the 
incoming VBV delay value, data should be output at a 
maximum rate. This places more frames of delay in the 
decoder buffer. This process continues until the VBV delay 
in the decoder buffer (i.e., VBV delay calculated in the PES 
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retiming circuit 300V) approximates the just-selected stream which might be evaluated include scene change indicia (e.g. , 

(To stream) VBV delay. The number of frames in the input a large number of intra-coded macroblocks), black field 

FIFO 220 is monitored during this process. If the FIFO 220 indicia, a minimal number motion, and the appropriate 

is nearing empty, a repeat frame is added to the stream from detection of anchor frames (i.e., I -frame followed by an 

the repeat frame store. When too much delay is in the 5 I-frame or a P-frame). By utilizing the external criteria, 

decoder buffer, output can be stopped until the proper frame adding and frame dropping may be performed in a 

amount of delay is in the decoder buffer, however it is manner which minimizes the impact on video and audio 

recognized this may be unacceptable in certain circum- quality for example. 

stances where there should be a continuous and substantially I n other words, frame synchronization is used to adjust the 

non-interrupted output of data as will be discussed. If FIFO data flow to accommodate differences between the input and 

220 is nearing overflow, a frame is dropped from the stream. output clock frequencies or to compensate for a shortage or 

When the VBV delays are aligned, output rate control over abundance of frames during the splice transition period, 

switches to a second mode of operation. Frame synchronization involves a decision to add or drop 

The second mode of operation is controlled by the PES frames and the mechanism for doing so. The decision to add 

retiming circuit 300V. The number of frames in the input J5 a frame is based upon the fullness of the switch input buffer 

FIFO plus the number of frames in the decoder buffer, when over a period of time. If the clock rate for a stream being 

added together and multiplied by the frame period produce switched to is lower than the output rate, the number of 

the end-to-end delay of this system. The end-to-end delay of frames in the input buffer will diminish over time and a 

a system may also be denned as the amount of time required repeat frame can be used to effectively add a frame to the 

for any individual byte to travel from the input of the receive 2Q mpu t buffer. Conversely a drop frame will remove a frame 

input FIFO to the output of the VBV buffer. This end-to-end from the input buffer that is rising in frame count. Frame 

delay value is stored for use in determining the remote dropping simply removes a frame which is preferably the 

source's clock timing difference from the local reference first B-frame after the last anchor frame in a GOP from the 

clock. The value is stored when the incoming VBV delay is indexed list of frames. A frame is added by inserting a 

aligned to the VBV delay in the decoder buffer. This 25 previously encoded P or B -type frame comprising uncoded 

end-to-end delay value is expected to remain constant when macroblocks with zero motion vectors. As set forth, such a 

the remote and local clocks are at exactly the same rate. frame acts as a repeat last frame command in an MPEG 

Alternately, the onset between the incoming PCR and the compliant system. As set forth however, this adding and 

local PCR can be stored. This offset is expected to remain dropping of frames changes the end-to end latency of the 

constant when the remote and local clocks are at exactly the 3Q overall system though. Accordingly, the present invention 

same rate. The PES retiming circuit extracts frames from the introduces latency control to the decision to add or drop 

FIFO based upon the frame rate embedded in the data frames. In a preferred form, overflow and underflow pre- 

stream. The decoder buffer contains the VBV delay currently vention is a higher priority than latency control, 

in the stream. Preferably, a fixed number of frames is maintained in the 

The controller 230 monitors the FIFO 220 and determines 35 preselected stream buffer FIFO 220. This buffer 220 may fill 

if the FIFO 220 is filling (i.e., remote clock faster than beyond this fixed number while waiting for an exit point thus 

station clock) or emptying (remote clock is slower than increasing the end-to-end delay or variable latency in an 

station clock). If the FIFO is tending to fill up then the undesirable manner. Further exasperating the situation as 

controller may responsively cause the add/drop unit 240 to further switches are completed this variable latency can 

drop one or more frames. If the FIFO is tending to empty 40 undesirably continue to grow or be reduced depending upon 

then the controller may responsively cause the add/drop unit the nature of the particular switch. Preferably the fixed 

240 to add one or more frames. This filling up or emptying number of frames is two or three frames, although a higher 

of the FIFO may occur in the second operating mode of the number of frames can of course be utilized 

frame synchronizer, when the FIFO is briefly storing each S o, after a switch from a "FROM" bitstream to a "TO" 

frame of the input ^video stream SI V and the PES switcher 45 bitst eam a B-frame is extracted once a GOP for example, to 

130 has selected the remote video and audio streams S2P V, return the number of frames stored in the bufcr * J he 

aZrA tor output. calculated number of frames, thereby advantageously 

Dropping frames is accomplished by choosing frames that decreasing the variable latency to an expected value asso- 

can be safely removed from the stream without upsetting dated with the fixed number of frames preferably stored. It 

either the picture re-ordering scheme or the predicted picture 50 should be recognized that the frame synchronization previ- 

quality. This may mclude dropping one or more frames to ously described preferably still functions to regulate decoder 

accomplish this end. Predicted picture quality is minimally buffer fullness, and may actually operate to reinstall some 

affected when the last B-frame before an anchor frame is "dummy" or repeat B-frames consistently with the method 

dropped. Any B frame may be deleted without affecting the earlier discussed to insure an underflow condition is guarded 

quality of the other pictures in the stream. 55 against. This may result in the latency control subsequently 

Adding frames is accomplished by inserting a P- or B- removing a later B-frame to continue to return the number 

type frame comprising uncoded macroblocks with zero of frames stored in the FIFO 220 to the fixed number, 

motion vectors. Such a frame acts as a "repeat last frame" however the action of the frame synchronizer 200 with the 

command in an MPEG-compliant syntactical manner as has latency control converges to maintain a fixed latency asso- 

been discussed. The added frame can be either a P-frame or 6 o ciated with the fixed number of frames over time thereby 

B-frame, though a P-frame is preferred because picture advantageously reducing the undesirable and unpredicted 

quality is best preserved when the added frame causes an variable latency to an expected value. 

I-frame to be repeated. The added "dummy" frame is stored Referring now also to FIG. 3, The VBV delay is modified 

using the controller 230 or add/drop unit 240 and inserted in the output stream to maintain buffer occupancy at the 

when necessary. 65 decoder. A model of a Standard Target Decoder (STD) Video 

Controller 230 may be used to monitor external criteria in Buffering Verifier is maintained as a Frame Byte Counter 

the input PES stream S1V. Examples of external criteria 354 for the output stream of the switch. This module tracks 
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the VBV Buffer fullness and VBV_delay of the output 
stream using the PCRB generated by the transport stream 
encoder (TSE) 150. A buffer size value (vbv_buffer^size_ 
value) is obtained from the sequence header. Each frame that 
leaves the TSE 150 contains a DTS that indicates the 5 
appropriate time to remove that frame from the VBV Buffer. 
When the PCRB equals the DTS that frame will be instan- 
taneously removed from the STD buffer. The VBV tracking 
module 354 accumulates the number of bytes sent within 
each frame and decrements the number of bytes in a given 
frame when the PCRB equals DTS for that frame. This can 10 
be accomplished by tracking the local PCRB, the number of 
bytes in each frame, and the DTS for each frame. Tests 
should be conducted to insure that the STD buffer neither 
overflows nor underflows. 

FIG. 3 shows a block diagram of a video PTS-DTS 1S 
retiming system 300 suitable for use in the systems of FIGS. 
1 and 2. The PTS-DTS retiming unit 300V receives a PES 
input bitstream S4P from a PES switcher 130 and delivers a 
PES output bitstream S7P to a TSE 150 in the studio or 
station case for example. The TSE performs the PCR stamp- 
ing at its output. The TSE feeds back a synchronized PCR 
base reference at a 90 KHz rate (the station clock of 27 MHz 
divided by 300). The PTS-DTS retiming unit operates as 
follows. 

25 

PCRB serial to parallel converter 312 converts the PCRB 
signal S9 from TSE 150 into a parallel (byte) stream S301 
which is coupled to an adder 314 and a subtractor 322. Adder 
314 adds the parallel PCRB S301 to a decoder delay time 
S304 to produce an adjusted PCRB time reference S302 30 
which is stored in an adjusted time reference register 316. 
The adjusted PCRB time reference S302 reflects the value of 
the PCRB count when a first picture may be removed from 
a video buffer and decoded at the far-end decoder. 

The addition of a decoder delay time S304 provides a 35 
mechanism for achieving a particular video buffer fullness 
level in a decoder of the type which uses the DTS for buffer 
control. The additional delay time causes the decoder to 
maintain a desired level of buffer utilization, e.g., fill video 
buffer with a certain number of pictures (perhaps all of the 40 
pictures representing the end-to-end-delay). One or more of 
the following factors may be used to determine the decoder 
delay time S304; the header entries of the VBV buffer size, 
the VBV delay and frame rate, the number of bytes in a 
frame and the number of frames (i.e., a frame byte count) in 45 
the decoder buffer. 

The method for using the delay time S304 to control the 
fullness of the decoder buffer is as follows. The adjusted 
PCRB is used as the DTS entry. When a video PES is present 
at the output of the video buffer at the decoder, the DTS 50 
(Decode Time Stamp) contained in the PES header is 
compared to the PCRB real-time clock reference derived 
from the PCR (Program Clock Reference) packets in the 
transport stream. The PES must await alignment of the DTS 
with the recovered PCRB. This alignment takes place after 55 
the period of time that the value of the delay time S304 
indicates, because the delay time S304 was added to the 
PCRB at the encoder, and this adjusted PCRB is used as the 
DTS entry. 

At a turn-on initialization time, an initialization command 60 
S306 causes a selector 318 to transfer the stored time 
reference S303 into a new time stamp register 320. Each 
time a PES header is decoded in the input bit stream S4P, an 
offset number, stored in the frame rate (FR) register 308, is 
added to the new time stamp S308 by an adder 310. The 65 
result of this addition is returned to the new time stamp 
register 320 via selector 318. 



The adder 310 offset number is specifically the number of 
cycles of 90 KHz contained in one frame time of the 
prevailing frame rate. The adder 310 offset numbers are 
shown in Table 1 (third column), and are automatically set 
as follows. A PES, sequence, pix and VBV delay decoder 
302 detects a sequence start code in the input data stream 
S4P and produces a sequence detect signal S312. A frame 
rate decoder 304 extracts the four-bit Frame Rate Code S310 
(first column of Table 1). The binary value of the frame rate 
code S310 corresponds to the frame rates listed in the second 
column of Table 1, as per the ATSC specification. A Look- 
Up-Table (LUT) 306 provides the offset number S311 (third 
column of Table 1) to frame rate register 308. In response to 
sequence detect signal S312, frame rate register 308 pro- 
vides the stored offset number to adder 310. It should be 
noted that since PES streams can have a plurality of 
sequences of intermixed frame rates, the PTS-DTS retiming 
unit automatically calculates all of the time stamp updates. 

The next step is to calculate the new PTS and DTS. DTS 
decode unit 328 decodes the DTS, if present, in the input 
data stream S4P to produce a DTS signal S321. DTS decode 
unit 328 also provides an indication flag S323 which is high 
(binary 1) if the input data stream S4P includes a DTS, and 
low (binary 0) if there is only a PTS (i.e., B-Frame 
operation). The DTS indication signal S323 is coupled to 
AND unit 334 and DTS update unit 340. The new DTS S327 
is taken from the new time stamp register 320 only if the 
DTS flag S323 indicates the presence of a DTS in the input 
data stream S4P. 

PTS decode unit 326 decodes the PTS in the input data 
stream S4P to produce a PTS signal S320. The difference 
S322 between the existing PTS S320 and DTS S321 is 
provided by subtractor 332 to AND unit 334. The new PTS 
S326 is calculated by adding the subtractor 332 output 
number S322 to the number S308 stored in the new time 
stamp register 320. This operation takes place in adder 336. 
Notice that when there is no DTS present (i.e., DTS Flag«=0), 
then the input S324 to adder 336 from AND unit 334 is zero 
(because of the operation of the AND unit 334) and new PTS 
S326 is the number S308 stored in the new time stamp 
register 320. 

The new PTS S326 and the new DTS S327 are stored in 
respective update registers 338340. A multiplex unit 344 
inserts the new PTS S326 and DTS S327 values into the 
output data stream S7P in place of the old values. A 
controller 343 tracks the location of the appropriate bits in 
the outgoing data stream S7P, and causes the multiplex unit 
344 to insert the updates in the correct positions in the 
stream. The outgoing data stream S7P proceeds to the 
transport stream encoder 150 in the usual manner. 

The PTS-DTS retiming unit 300V also provides the 
ability to manage end-to-end delay in the system by account- 
ing for buffer behavior in a far-end decoder (e.g., a set-top 
receiver) and buffer or bitstream generation behavior in the 
bitstream sources (e.g., encoder 210 and server 220), as will 
now be described. 

PES, sequence, pix and VBV delay decoder 302 detects 
the start of a PES packet in the input data stream S4P and 
produces a PES detect signal S3 13 which is coupled to the 
new time stamp register 320. At each PES interval, the new 
time stamp register 320 is updated with a fixed constant 
S318 which is determined by the prevailing frame rate. The 
updated contents S308 of the new time stamp register 320 
are compared with the contents S303 of the adjusted time 
reference register 316 by a comparator 323. The comparator 
result (i.e., less than, greater than, or equal to) provides an 
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indication of the rate at which the PES data is being 
processed (i.e., too slowly, too quickly, or just right). If the 
PES streams S4P are being issued too slowly by, e.g., the 
live video encoder 210 or server 220, the encoder 210 or 
server 220 output buffers (not shown) may be read or 
emptied at a maximum rate. If the comparison indicates that 
processing is proceeding too quickly, the rate of reading the 
buffers in the encoders or servers can be reduced, A buffer 
control signal S12 may be used to control encoder 210 or 
server 220 buffer utilization. 

VBV buffer fullness is monitored to properly control the 
flow of the video stream to the decoder. At initialization (i.e., 
initialization command S306 asserted), the number of bytes 
output from the PTS-DTS retiming unit 300V are stored on 
a frame by frame basis. The current time, decode time for 
each frame, the number of bytes per frame, and the VBV 
buffer size are used to determine the fullness of the VBV 
buffer. At initialization the VBV buffer is empty. As the first 
frames are output to the VBV buffer, a running total of the 
bytes in the VBV buffer is maintained by a frame byte 
counter 354. If this total equals the VBV buffer size, an 
output signal S317 from the frame byte counter 354 to the 
logic unit 325 causes the logic unit 325 to prevent data from 
being output from the PTS-DTS retiming unit 300V. 

Two modes of VBV buffer operation are possible as 
described in Annex C of the MPEG2 specification. The first 
mode (Mode A) requires VBV delay to be set to hexadeci- 
mal FFFF. The second mode (Mode B) requires the VBV 
delay to not be set to hexadecimal FFFF, but instead reflect 
the true value of VBV delay in the VBV buffer. During 
Mode A initialization, bytes are output to the VBV buffer 
until the buffer is full as indicated by frame byte counter 354. 
Once the VBV buffer is full, data is extracted from the VBV 
buffer as described in Annex C of the MPEG2 specification 
which is illustrated in Table- 1 hereof. During Mode B 
initialization, bytes are output to the VBV buffer until the 
calculated value of VBV delay is greater than or equal to the 
VBV delay value in the incoming stream S4P. The incoming 
stream should be self consistent and MPEG2 compliant in 
that the buffer will not over/under flow using the values 
contained in the stream. The calculated value of VBV buffer 
fullness contained in frame byte counter 354 is used to 
prevent under/over flow of the VBV buffer. Data flow to the 
VBV buffer can be regulated using this calculated value if 
the incoming stream for some reason is not consistent. The 
DTS-PTS values in the stream should be consistent with the 
VBV delay values contained in the stream. 

When switching to a stream in Mode A, the hexadecimal 
FFFF value of VBV delay in the next-selected stream is 
overwritten with the calculated value maintained in the 
PTS-DTS retiming section. When switching to a stream in 
Mode B, the hex value of VBV delay in the next-selected 
stream is overwritten, when necessary, with the calculated 
value maintained in the PTS-DTS retiming section 300V 
until the calculated VBV buffer value and the next-selected 
stream VBV delay value are aligned. Otherwise the incom- 
ing stream data flow is held until the VBV delays align. This 
process takes advantage of the re-multiplexing operation 
that is occurring in the TSE that follows the retiming circuit. 
During the alignment period, data may be output to the VBV 
buffer at maximum video bit rate, to increase the VBV delay 
in the VBV buffer. This process can continue until the VBV 
delays align as long as the buffer does not become full. 

A key element in implementing the present splicing 
technique is regulating the flow of data through the transport 
multiplex. The flow of video data is regulated to insure 
neither an overflow nor an underflow of the VBV buffer, and 
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the STD Video Buffer Verifier Delay (VBV__delay) values 
in the output stream are maintained at levels consistent with 
the relationship between Program Clock Reference (PCR) 
and Decode Time Stamp (DTS) while neither over- nor 
under-flowing the VBV Buffer. 

The objective of flow control is to re -align the delay 
associated with the VBV buffer (vbv_delay) with the vbv_ 
delay in the incoming stream during the transition period 
that follows a stream splice. If the actual vbv_delay, as 
indicated by the STD Video Buffering Verifier module 354 
is larger than the incoming stream vbv_delay, the video 
flow to the output can be slowed until the vbv_delays match 
as opposed to stopping the flow of data as has been previ- 
ously discussed. When the actual vbv_delay is smaller than 
the incoming vbv__delay, the flow control operates to output 
the video faster than originally planned by the incoming 
stream. Again though, STD buffer fullness levels are still 
tested to prevent buffer overflow or underflow as has been 
discussed. 

The flow rate is calculated on a frame by frame basis to 
assure video data is output at a rate commensurate with the 
original stream. The vbv_delay of the first frame after the 
splice is set to the actual vbv_delay being tracked by the 
STD video buffer verifier module. The vbv_delay of the 
next frame in the input buffer becomes the target value for 
vbv_delay. The flow rate is calculated by dividing the 
number of bytes in the first frame by a time interval. The 
time interval is the actual vbv_delay minus the target 
vbv_delay plus a frame rate factor all expressed in seconds. 
The frame rate factor is found from a lookup table based 
upon sequence header values for progressive and low delay 
as shown in Appendix C of 13818-1 MPEG Systems Speci- 
fications and as is identified by Table 1. 

TABLE 1 

FRAME RATE 



CODE IN 


FRAME RATE 




ADDER 310 


SEQUENCE 


VALUE 




OFFSET 


HEADER 


(Hz) 




NUMBER 


0001 


24/1.001 (23.967 . 


■-) 


3753.75 


0010 


24 




3750 


0011 


25 




3600 


0100 


30/1.001 (29.97 . . 


•) 


3003 


0101 


30 




3000 


0110 


50 




1800 


0111 


60/1.001 (59.94 . . 


•) 


1501.5 


1000 


60 




1500 



Once the flow rate is known, actual flow control is prefer- 
ably performed at the packet rate to keep computational 
complexity to a minimum. The time interval, TI, available to 
send a frame of data to a decoder buffer can be represented 
by: 



(1) 



where T(n) is the proper vbv_delay in seconds for frame n, 
T (n+1) is the proper vbv__delay in seconds for a frame n+1 
immediately succeeding frame n, and FP is the period of 
time between successive examinations of the vbv buffer as 
detailed in Annex C of the MPEG specification. The maxi- 
mum rate that bytes can leave the flow control buffer equals 
the maximum transport stream rate minus the rate of audio 
and system data (including program specific data rate) 



Rm ax-Transport Byte Rate-AUDIOrate-SYSTEMrate (2) 
dm# bytes in next frame in the Flow Control Buffer (3) 
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Rflow=J/77 (4) determining a time interval in which to transmit at least 

if n<=Q t then Rftow-Rmax (5) one of the frames; and, 

if Rflow>-Rmax, then Rflow-Rmax (6) utilizing said time interval to determine said flow rate. 

, j + j *l. j • t_ , 9. The method ofclaim 8, wherein each said time interval 

It should also be understood that considermg that the 5 ^ defined as T(n) . T(n+1)+FP> where T(n) is of a 

disclosed splicer parses TO and FROM bitstreams, it is time associated with a frame n, t(n+l) is a time associated 

entirely possible to convert a Constant Bitrate (CBR) bit- with a frame n+1, frame n+1 immediately follows frame n 

stream to a Variable Bitrate (VBR) bitstream by detecting in said TO bitstream and FP is a period of time between 

the padding and stuffing needed to create a CBR. Removing successive examinations of said buffer. 

the padded bytes gets rid of unwanted overhead, and makes 1Q 10. The method of claim 1, further comprising: 

room for revenue producing opportunistic data insertion in identifying portions of said data in said TO or FROM 

the remultiplexing. bitstream associated with stuffing or padding data; and, 

Although the invention has been described and pictured in substituting opportunistic data for said identified data, 

a preferred form with a certain degree of particularity, it is 11. The method of claim 10, wherein said opportunistic 

understood that the present disclosure of the preferred form, data is indicative of additional information transmitted in 

has been made only by way of example, and that numerous said system. 

changes in the details of construction and combination and 12. A transport layer switching system suitable for use in 

arrangement of parts may be made without departing from a compressed domain digital communications system sub- 

the spirit and scope of the invention as hereinafter claimed. J ect to experiencing a variable end-to-end delay at least 

It is intended that the patent shall cover by suitable expres- 2Q partially resulting from at least one splice between a FROM 

sion in the appended claims, whatever features of patentable bitstream and a TO bitstream each respectively including 

novelty exist in the invention disclosed. data corresponding to a plurality of frames, said transport 

We claim: layer switching system comprising: 

1. In a compressed domain digital communications a first buffer for storing a portion of said TO bitstream data 
system, a method for reducing a variable latency associated 25 being indicative of said end-to-end delay; and, 

with a buffer and at least partially resulting from at least one a controller coupled to said first buffer, wherein said 
splice between a FROM bitstream and a TO bitstream each controller is responsive to an increase in said end-to - 
including data corresponding to a plurality of frames, the end delay by removing data associated with at least one 
method comprising: select frame from said first buffer, to a possible over- 
temporarily storing a portion of said TO bitstream data in 30 flow condition associated with a second buffer down- 
a buffer; stream from said switching system by reducing a rate of 
determining a value associated with said buffered data and transmission for a portion of said data associated with 
being indicative of said variable latency; at least one of said frames, and to a possible underflow 
regulating a flow of data in the system to reduce the condition associated with said second buffer by effect- 
variable latency when the determined value corre- 35 ing a repeat last frame command using said first buffer, 
sponds to a number of frames greater than a given 13. The system of claim 12, wherein the communications 
number of frames by selectively deleting a portion of system is ATSC digital television standard compliant and 
said buffered data corresponding to a select at least one eactl ^ GCi frame is a B-frame. 

of the frames from the buffer; and, The system of claim 13, wherein each select B-frame 

regulating a flow of data in the system to prevent an 40 corresponds to a distinct group of pictures, 

underflow condition in the system by effecting a repeat 15 * ^ s y stem of claim n > wherein the communications 

last frame command using said buffered data and system is an ATSC digital television standard compliant 

prevent an overflow condition in the system by slowing system and the step of effecting a repeat last frame command 

a rate of transmission for at least a portion of the data comprises inserting a previously encoded P- or B-type frame 

associated with at least one of the frames in the TO 45 mcludin g encoded macroblocks with zero motion vectors, 

bitstream while maintaining a substantially noninter- 16 ^ of claim 12 > wherein said reducing said 

rupted flow of data in said system. rate °^ transmission for at least one of the frames comprises 

2. The method of claim 1, wherein each select frame is calculating a flow rate on a frame by frame basis, 
selected so as not to significantly effect visual play-out of the 17 ' ^ s y stem of claim 16, wherein said calculating a 
TO bitstream. 50 ^ ow rate on a f rame by frame basis comprises: 

3. The method of claim 1, wherein the system is ATSC determining a time interval in which to transmit said data 
digital television standard compliant and each select frame associated with one of the frames, respectively; and, 
is a B-frame. utilizing said time interval to determine a flow rate. 

4. The method of claim 3, wherein each select B-frame 18. The system of claim 17, wherein said time interval is 
respectively corresponds to a distinct group of pictures. 55 defined as T(n)-T(n+l)+FP, where T(n) is indicative of a 

5. The method of claim 1, wherein the regulating effec- time associated with a frame n, t(n+l) is a time associated 
tively reinserts a select frame deleted. with a frame n+1, frame n+1 immediately follows frame n 

6. The method of claim 1, wherein the system is an ATSC m sa id TO bitstream and FP is a period of time between 
digital television standard compliant system and the step of successive examinations of said buffer. 

effecting a repeat last frame command comprises inserting a 60 1^- The system of claim 12, further comprising: 

previously encoded P or B-type frame including uncoded identifying portions of said data in said to or FROM 

macroblocks with zero motion vectors. bitstream associated with stuffing or padding data; and, 

7. The method of claim 1, wherein said slowing said rate substituting opportunistic data for said identified data, 
of transmission for at least one of the frames comprises 20. A method for switching from a first bitstream to a 
calculating a flow rate on a frame by frame basis. 65 second bitstream both including data, in a compressed 

8. The method of claim 7, wherein said calculating a flow domain digital television transmission system, the method 
rate on a frame by frame basis comprises: comprising: 
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identifying an entry point in said second bitstream; 

buffering said second bitstream data dependently upon 
said identified entry point, said buffered data corre- 
sponding to a number of video frames; 

determining if said number of video frames exceeds a 5 
given number, and if so removing a portion of said 
buffered data corresponding to at least a select one of 
said frames; 

determining whether said buffered data is likely to cause 
an overflow condition in a downstream buffer when 
transmitted to said downstream buffer, and if so slow- 
ing a rate of transmission associated with a portion of 
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said buffered data corresponding to at least one of said 
frames while ensuring a substantially non-interrupted 
flow of data in said system to said downstream buffer; 
and, 

determining whether said buffered data is likely to cause 
an underflow condition in said downstream buffer when 
transmitted to said downstream buffer, and if so effec- 
tuating a repeat frame command dependent upon at 
least a portion of said buffered data. 

***** 
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